There are several common procedures typically used in the creation of visual effects for cinema and broadcast. These include “matchmoving”, the term that refers to the process of matching the position and angle of a visual effect (e.g., object) to be inserted in a live-action footage and the live-action footage itself. Examples of visual effects include computer generated imagery (e.g. animated creature), a soccer off-side line, distance between elements on field, highlight on various elements in video, an advertisement logo, graphic sequences build from still frames, 3D objects rendering, insertion of external video source as visual effects, and any other object insertion and/or modification.
Live-action footage can be shot with a film, video or television camera. The matchmoving procedure is a required step for creating a plausible looking visual effect. Applications utilizing the matchmoving include altering the appearance of some objects intended for advertising some brand or company, in a movie, which is carried out in accordance with a geographical location of the movie audience. Also matchmoving is used for adding auxiliary lines or symbols to sport broadcasts, such as lane numbers looking as if they were under the water in swimming competitions, or off-side lines in soccer games. Yet other matchmoving-based applications use the addition of animated characters to a movie or, vice versa, addition of real actors to computer generated scenes, etc.
In matchmoving applications, there are generally two main approaches to getting information about the shooting camera parameters, necessary for an accurate adjustment of the graphical objects to the video stream. The first approach is based on the use of sensing systems, targeted to provide the camera parameters through physical measurements. The sensor-based systems are characterized by their universality, meaning these systems can be used independently, irrespective of the content of the video stream. On the other hand, they suffer from typical limitations of measuring systems, such as insufficient sensing accuracy (especially for outdoor conditions), time-consuming calibration procedures, and high cost.
The second approach widely exploits computer vision techniques, according to which required information about the camera parameters is extracted directly from the video stream. This class of methods has been intensively investigated in the past 15 years.
The procedure of extraction of information about the camera parameters (e.g., position and orientation in three dimensions, focus, zoom, distortion) from a footage is known as camera calibration for a fixed camera or camera tracking for a moving camera. The determination of the camera parameters may be facilitated in the presence of a camera sensing system in case of combination of the two aforementioned general approaches. Camera tracking is used in the U.S. Pat. Nos. 6,100,925 and 5,436,672 as well as in the automated matchmoving software products Realviz MatchMover and Boujou, commercially available respectively from RealViz S. A. and 2d3 Limited. However, this procedure is fairly complex due to the necessity to determine three-dimensional (3D) parameters from two-dimensional (2D) frames. The major difficulties in the camera tracking techniques are associated with the necessity to predefine a set of calibration features and their geometrical relationship; the necessity to satisfy time constraints, especially stringent for live broadcasts; the potential loss of the tracked calibration features due to rapid motions, camera occlusions, or poor imaging conditions. When using a sensing system, targeted to provide the camera parameters through a physical measurement, the difficulties are associated with insufficient accuracy of sensing, as well as increased product price and the need for additional equipment.
Another approach utilizes the so-called scene motion (global motion) tracking technique, which is advantageous over camera tracking in that it is capable of evaluating direct frame-to-frame (field-to-field) geometric transformations by image processing algorithms. Examples of scene motion tracking methods include block matching, optical flow, and other techniques known in the art [Y. Wang, J. Ostermann, Y. Zhang, “Video processing and communications”, Prentice Hall, 2001, pp. 141-216; and U.S. Pat. No. 5,808,695]. Most known techniques of this type utilize a selected set of reference points/features to find transformations between the successively grabbed frames.
Examples of the above methods can be found in the following U.S. Pat. Nos. 5,264,933; 5,353,392; 5,515,485; 5,731,846; 5,892,554; 5,491,517; 5,436,672; 6,100,925; 6,181,345; 6,765,569.